Live streaming with live video production and commentary

ABSTRACT

A method comprises receiving from each of a plurality of commentator applications corresponding commentary information relating to video content from at least one video source, sending at least portions of the commentary information received from each of the commentator applications to one or more other ones of the commentator applications, and generating commented video content based at least in part on the commentary information received from the commentator applications. The commented video content is provided to one or more servers of a content delivery network for delivery to one or more viewer devices. The receiving and sending are illustratively implemented in a media server through interaction of the media server with web browsers of respective commentator devices that implement the commentator applications. Each of the web browsers may implement an instance of a commentary mixer configured to combine commentary information from its commentator application with additional commentary from other commentator applications.

RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 16/225,335, filed Dec. 19, 2018 and entitled “LiveStreaming with Multiple Remote Commentators,” which claims priority toU.S. Provisional Patent Application Ser. No. 62/719,278, filed Aug. 17,2018 and entitled “Live Streaming with Multiple Remote Commentators,”each incorporated by reference herein in its entirety. The presentapplication also claims priority to U.S. Provisional Patent ApplicationSer. No. 62/883,732, filed Aug. 7, 2019 and entitled “Live Streamingwith Live Video Production and Commentary,” which is also incorporatedby reference herein in its entirety.

FIELD

The field relates generally to live video and other types of mediacontent, and more particularly to processing of media content.

BACKGROUND

The rapidly growing use of mobile devices such as laptops, tablets andcellphones has greatly diversified the available modes of mediaconsumption. In these and other contexts, a wide variety of differentmedia streaming techniques are known, including techniques for streamingof media over the Internet using hypertext transfer protocol (HTTP).More specific examples of HTTP streaming techniques include the AppleHTTP Live Streaming (HLS) protocol, Microsoft Smooth Streaming (MSS),and MPEG Dynamic Adaptive Streaming over HTTP (DASH). Various non-HTTPstreaming techniques are also known, including real-time messagingprotocol (RTMP). Despite recent advances in this area, a need remainsfor improved techniques for streaming of live video and other types ofmedia content.

SUMMARY

Illustrative embodiments of the invention provide content deliverysystems with functionality for live video streaming augmented with audiocommentary or other types of commentary information from one or moreremote commentators. These and other embodiments additionally oralternatively include functionality for live video streaming with livevideo production.

The growing capacity of the Internet to accommodate streaming media hasnaturally led to rapid increases in the number of live events that canbe viewed online. Both major and niche events can now reach audienceseverywhere with Internet connectivity. For example, a live event, suchas a sporting event or an e-gaming event, can be watched by many peoplein different countries or other geographic regions around the world,possibly with different commentary audio provided to different audiencesin the local language of their respective countries or other geographicregions.

In conventional practice, such arrangements generally require havingmultiple commentators fluent in various languages physically presenteither at the event or in television studios equipped with specializedtelevision studio equipment. The specialized television studio equipmentmay include, for example, specialized digital video processing hardwareused to keep voice from one or more of the commentators and video fromthe live event synchronized in a broadcast to a given audience.

Unfortunately, these conventional approaches are unduly expensive, inthat they require the commentators to travel either to the actual eventor to a television studio, and also require the above-noted specializedtelevision studio equipment. Such conventional approaches can beprohibitively expensive for many less affluent markets, and moreover donot readily scale to large numbers of commentators.

Illustrative embodiments disclosed herein solve these and other problemsof conventional approaches, for example, by allowing multiple remoteregistered commentators, equipped with widely-available personalcomputing devices, such as personal computers, tablets, or smartphones,suitably modified with respective commentary applications, to addcommentary over the Internet to existing live streaming video and audiobroadcasts.

In some embodiments, video is generated at a live event and there areone or more registered remote commentators in respective differentlocations remote from the live event each of whom generates audio orother commentary information that is combined with the video of the liveevent.

A media processor is illustratively configured in accordance withtechniques disclosed herein to ensure that the audio or other commentaryinformation from the remote commentators and the video from the liveevent are correctly synchronized and mixed before the combined contentis segmented and distributed to end users. Such end users are alsoreferred to herein as “viewers.”

In one embodiment, a method comprises receiving from each of a pluralityof commentator applications corresponding commentary informationrelating to video content from at least one video source, sending atleast portions of the commentary information received from each of thecommentator applications to one or more other ones of the commentatorapplications, and generating commented video content based at least inpart on the commentary information received from the commentatorapplications. The commented video content is provided to one or moreservers of a content delivery network for delivery to one or more viewerdevices, and illustratively represents a commented version of videocontent that comprises live video from at least one live video source.

The commentary information received from each of the commentatorapplications illustratively comprises respective distinct streams ofmedia content from respective ones of the commentator applications, withthe stream of media content from a corresponding one of the commentatorapplications comprising at least one of audio content, video content,image content, social media posting content, chat text and closedcaption text.

The commentator applications in some embodiments receive respectiveinstances of a relatively low resolution version of the video contentfrom at least one video source. The relatively low resolution version ofthe video content is illustratively generated in a pre-mixer of said atleast one processing device, utilizing a plurality of relatively highresolution content streams from respective ones of a plurality of videosources.

In some embodiments, the receiving and sending are implemented in amedia server of said at least one processing device through interactionof the media server with respective web browsers of the respectivecommentator devices.

Each of the web browsers illustratively implements an instance of acommentary mixer configured to combine commentary information from itscorresponding commentator application with additional commentaryinformation received from respective other ones of the commentatorapplications via the media server.

The instances of the commentary mixers implemented by respective ones ofthe web browsers in some embodiments are synchronized with one anotherrelative to the video content to less than a specified amount of delayin order to support apparent real-time interaction between users of thecommentator applications in the commented video content as viewed at theone or more viewer devices.

Additionally or alternatively, a first web browser of a first one of thecommentator devices is illustratively configured to present commentaryinformation received from respective other web browsers of other ones ofthe commentator devices via the media server. The commentary informationreceived from the respective other web browsers via the media server ispresented by the first web browser in respective distinct displaywindows, or using other browser-based display techniques.

Although web browsers are used in some embodiments, a wide variety ofother types of application programs, such as native desktop applicationsor other computer applications that do not operate as or otherwiseinclude web browsers, can be used in addition to or in place of webbrowsers to support remote commentary in other embodiments.

In some embodiments, generating the commented video contentillustratively comprises generating the commented video content bycombining, in a post-mixer of said at least one processing device, atleast portions of the commentary information received from thecommentator applications with a relatively high resolution version ofthe video content from said at least one video source.

Illustrative embodiments are advantageously configured to readilyaccommodate large numbers of remote commentators, such as remotecommentators that are located in close enough proximity to one anotherso as to have sufficiently low voice communication delay between them.For example, multiple groups of such remote commentators can be presentin respective different countries or other geographic regions with eachsuch group providing audio commentary for the content delivered to thatcountry or other geographic region. Numerous other arrangements arepossible in other embodiments.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an exemplary content delivery system with acloud-based remote commentator configuration in an illustrativeembodiment.

FIG. 2 is a flow diagram of a mixer algorithm in an illustrativeembodiment.

FIG. 3 is a block diagram of a content delivery system with multipleremote commentators in an illustrative embodiment.

FIG. 4 is a flow diagram of a mixer algorithm for multiple remotecommentators in an illustrative embodiment.

FIG. 5 is a block diagram of a cloud-based commentator system in anillustrative embodiment.

FIG. 6 is a block diagram showing a set of mixers and a media server ina cloud-based commentator system in an illustrative embodiment.

FIG. 7 is a block diagram showing synchronization of the set of mixersand the media server in the FIG. 6 cloud-based commentator system usinga layout data store in an illustrative embodiment.

DETAILED DESCRIPTION

Embodiments of the invention will be illustrated herein in conjunctionwith exemplary content delivery systems that include particulararrangements of networks, devices and other components. It should beunderstood, however, that embodiments of the invention are moregenerally applicable to a wide variety of other types of contentdelivery systems and associated networks, devices or techniques. Theterm “content” as used herein is intended to be broadly construed so asto encompass, for example, live video or other types of multimediastreams as well as other types of content that are deliverable todevices over one or more networks in a content delivery system.

Illustrative embodiments include but are not limited to methods,apparatus, systems, processing devices, integrated circuits, andcomputer-readable storage media having computer program code embodiedtherein.

Some embodiments are configured to utilize streaming techniques that arebased at least in part on the above-noted Apple HLS protocol. However,it is to be appreciated that other embodiments can be configuredutilizing a wide variety of other types of streaming protocols andaccordingly are not limited to use with live streaming or HTTP.Accordingly, illustrative embodiments can utilize other HTTP streamingtechniques such as MSS and MPEG DASH, in addition to or in place ofApple HLS. Non-HTTP streaming techniques such as RTMP can also be used.

Additionally or alternatively, some embodiments are configured toutilize techniques disclosed in one or more of U.S. Pat. No. 9,635,431,entitled “Live Event Viewing via Mixed Live and On-Demand Streaming,”U.S. Pat. Nos. 10,182,270 and 9,654,844, both entitled “Methods andApparatus for Content Interaction,” U.S. Pat. Nos. 9,661,355 and9,832,491, both entitled “Virtual Immersion Via Streamed ContentAdaptation,” U.S. Pat. Nos. 10,419,513 and 9,900,362, both entitled“Methods and Apparatus for Reducing Latency Shift in Switching BetweenDistinct Content Streams,” and U.S. Pat. No. 9,942,343, entitled“Efficient Content Streaming Utilizing Local Proxy Server Implemented onClient Device,” each of which is incorporated by reference herein in itsentirety. It is to be appreciated, however, that utilization of suchtechniques is not a requirement in any particular illustrativeembodiment.

FIG. 1 shows a content delivery system 100 that implements functionalityfor one or more remote commentators, although a commentator station ofonly a single remote commentator is explicitly shown in the figure. Inthis embodiment, the content delivery system 100 illustrativelycomprises a live media server 102 that comprises a video and surroundaudio encoder 103. The live media server 102 receives live video from avideo camera 104 that is assumed to be arranged to capture video of alive event.

The live media server 102 is coupled to a network 105 that includes amedia processor 106 and a plurality of content delivery network (CDN)web servers 108-1, . . . 108-k, . . . 108-m. The media processor 106provides HLS streams including media segments and associated playliststo the CDN web servers 108. The CDN web servers 108 deliver contentstreams to respective client devices of respective viewers responsive torequests received from those client devices. Each such client deviceimplements a media player for requesting and playing content for itscorresponding viewer. The client devices of the respective viewers arenot explicitly shown in the figure, but can include various types ofmobile devices.

The playlists of the HLS streams may be illustratively implemented asrespective “index files,” although other types of playlists can be usedin other embodiments. Such an index file or other type of playlist insome embodiments illustratively provides an ordered list of the UniformResource Locators (URLs) of the corresponding media segments. Othertypes of media segment identifiers can be used in other embodiments.

Also coupled to the network 105 is a registered commentator computingstation 110-1, illustratively implemented as a cellphone, also referredto as a mobile telephone or a “smartphone,” having a screen 112 and amicrophone 114. The registered commentator computing station 110-1,which is also referred to herein as simply a “commentator station,” maybe viewed as an example of what is more generally referred to herein asa “registered commentator device.” A wide variety of other types ofregistered commentator devices can be used, including various othertypes of mobile devices, client devices or other types of personalcomputing devices, such as personal computers or tablets.

The commentator station 110-1 implements a commentator application 115that is configured to interact with the media processor 106 over thenetwork 105 in providing remote commentary for live video. More detailedexamples of such interaction will be described below in conjunction withthe flow diagrams of FIGS. 2 and 4. The commentator application 115 isillustratively implemented at least in part as one or more softwareprograms stored in a memory of the commentator station 110-1 andexecuted by a processor of the commentator station 110-1. The one ormore software programs when executed provide functionality fordelivering remote commentary from the commentator station 110-1 back tothe media processor 106, at least in part responsive to video receivedfrom the media processor 106 and displayed on the screen 112 of thecommentator station 110-1.

The remote commentator associated with the commentator station 110-1provides audio input to the commentator application 115 via themicrophone 114. Other types of input can be provided using other userinput mechanisms. For example, touch input can be provided via thescreen 112. Other user input mechanisms, such as a mouse or keyboard,can be used in addition to or in place of the touch-screen inputmechanism. It is also possible that user input can be provided to thecomputing station via spoken commands or gestures, which are recognizedby respective speech recognition or gesture recognition functionalityimplemented in the commentator station 110-1. Various combinations ofthese and other user input mechanisms can be used in a given embodiment.

A user of the commentator station 110-1 is also referred to herein inthe context of some embodiments as a “registered commentator.” Variousauthentication mechanisms can be used in illustrative embodiments inorder to ensure that only commentators presenting the appropriatecredentials can access the commentator application 115 on thecommentator station 110-1.

The commentator application 115 in this embodiment is assumed tocomprise a video streaming application, suitably adapted to support theremote commentator functionality disclosed herein. Other modulesproviding other functionality can also be implemented within theapplication. Moreover, the commentator station 110-1 can incorporatemultiple applications, although only a single application is shown inthe present embodiment. In some implementations, the commentatorapplication 115 can comprise portions of multiple applications.Accordingly, the term “application” as used herein is intended to bebroadly construed. Such an application is also referred to herein as an“application program” although it is to be appreciated that anapplication program can itself comprise multiple distinct softwareprograms.

The commentator station 110-1 is just one example of a client device. Itis to be appreciated that a wide variety of different media players orother client devices can be used, and such media players or other clientdevices need not be implemented using a built-in HLS client arrangement.For example, other types of built-in clients can be used. Thus, a“client device” as that term is broadly used herein should not beconstrued as being limited, for example, to a hardware-assisted mediaplayer that utilizes a client built into the media player operatingsystem. Accordingly, in other embodiments, a client device can includean internal media player. The built-in HLS client can itself beconsidered a type of media player.

The CDN web servers 108 in the FIG. 1 embodiment are examples of whatare also referred to as distributed HTTP based web servers or simply asdistributed web servers. Such servers can be configured to deliver awide variety of media content other than live video. The CDN web serversin some embodiments comprise a collection of distributed web serversthat are set up in a cloud or other type of network such as network 105in order to distribute live video or other types of media content.Numerous other arrangements of distributed web servers can be used inother embodiments.

The media processor 106 may comprise one or more video servers, and mayalso be referred to as a content provider server.

The network 105 over which the live media server 102, the commentatorstation 110-1, the media processor 106 and the CDN web servers 108communicate is assumed to support HTTP communications. It should benoted that, although HTTP communications are used in the presentembodiment, other embodiments can utilize other types of protocols formedia streaming over the Internet, or more generally any of a widevariety of other techniques for media content delivery. At least aportion of the CDN web servers 108 may be part of a cloud arrangement.

Each of the CDN web servers 108 is configured for media streaming. Eachsuch web server illustratively caches video segments and associatedindex files received from the media processor 106 over the network 105.

The content delivery system 100 can include multiple instances ofcomponents such as live media server 102, video camera 104 and mediaprocessor 106, although only single instances of such components areshown in the figure for clarity and simplicity of illustration.

Media segments and associated index files are supplied by the mediaprocessor 106 to at least a subset of the CDN web servers 108 over thenetwork 105 via one or more connections. The encoding of video inillustrative embodiments can utilize known encoding techniques such asH.264. Also, the segmenting of the encoded video can be performed inaccordance with known streaming protocols such as Apple HLS, MSS or MPEGDASH.

It is to be appreciated, however, that a wide variety of differentencoding and segmenting techniques can be used in other embodiments,including, by way of example only, those techniques described in theabove-cited U.S. Pat. Nos. 10,419,513 and 9,900,362.

Each of the CDN web servers 108 illustratively stores multiple indexfiles as well as sets of video segments associated with respective onesof those index files. As noted above, index files are consideredexamples of what are more generally referred to herein as “playlists.”The video segments are considered an example of what is more generallyreferred to herein as “media segments.” A wide variety of differentarrangements of index files or other types of playlists, and associatedvideo segments or other types of media segments, can be used indifferent embodiments.

For example, in some embodiments, live video can be streamed within thecontent delivery system 100 utilizing HTTP streaming technology such asthe above-noted HLS, MSS or MPEG DASH protocols. With HTTP streaming,video associated with a given content stream is segmented by the mediaprocessor 106. As soon as a given video segment is ready, it isdelivered to the CDN web servers 108 so as to be available for massdistribution to client devices of respective viewers within the system100. At session initiation, a media player obtains an initial masterplaylist indicating the available content streams and their associatedweb addresses (e.g., URLs). Depending on the streaming technology,locating the available content streams may be an indirect process wherethe master playlist points to index files that can be polled to indicatethe URL and availability of the next segment.

Media streaming using HTTP based protocols has become ubiquitous due toits flexibility, compatibility with generic web servers such as CDN webservers 108 for content distribution, and ability to traverse commonfirewalls. HTTP streaming standards such as Apple HLS generally work bybreaking the content stream into small HTTP-based file segments, whichare distributed to the CDN web servers 108 and downloaded by request bya media player client via each segment's uniquely assigned web address(e.g., URL).

In some embodiments, HLS streaming relies on playlists that contain theURLs of the available media segments. These playlists reside in the sameCDN web servers 108 with the media segments to be streamed. At sessioninitiation, the media processor 106 downloads a master playlistcontaining the URLs of the various alternative playlists available forthe desired content. Mostly, the optional playlists allow the player tooptimize playback based on screen resolution and bandwidth availability.Once given the playlist URLs, a built-in HLS client can autonomouslydownload the referenced playlist, request media segments, thenreconstruct and play the requested video stream.

In addition to the above-noted bandwidth options, HLS supports differentcontent specific playlists, including live playlists, event playlists,and video-on-demand (VoD) playlists, as described in Apple TechnicalNote TN2288, which is incorporated by reference herein. The VoDplaylist, which is used for playback of prerecorded media, containsreferences to all the media segments for the video. The client needs todownload the list only once at the start of a session. On the otherhand, both live and event types of broadcast require continuous updatesto their respective playlists as new video segments are created anduploaded to the CDN web servers. As such, the client must alsorepeatedly download the referenced playlist to get the latest mediasegment URL.

The operation of the media processor 106 and its interaction withcommentator application 115 of the commentator station 110-1 to supportremote commentator functionality will now be described in more detail.

In one possible operating scenario, the media processor 106 receivesvideo content from the live media server 102. The video content includesvideo of a live event as captured by the video camera 104 and mayadditionally include associated surround audio from multiple microphonesdeployed at the live event. The video content is delivered from the livemedia server 102 to the media processor 106 via the network 105,illustratively using a high-speed connection based on a protocol such asRTMP or web real-time communications (WebRTC).

The media processor 106 includes a timestamp module 120 that generatestimestamps for respective frames of the video content. A given suchtimestamp is denoted Tm in the figure, and is an example of what is alsoreferred to herein as a “first timestamp.” The timestamp module 120timestamps incoming frames of the video content to generate respectivefirst timestamps.

The timestamped video content is duplicated or otherwise split as shown,to produce two identical streams, with a first one of the streams beingapplied to a media buffer 121 of the media processor 106, and a secondone of the streams being delivered to the commentator application 115 ofthe commentator station 110-1 over the network 105. The second stream isillustratively delivered to the commentator station 110-1 again using aprotocol such as RTMP or WebRTC, but possibly at a slower speed thanthat used by the high-speed connection between live media server 102 andmedia processor 106.

A timestamp module 122 in the commentator application 115 is configuredto associate frames of audio content comprising audio input receivedfrom the remote commentator via microphone 114 with respectivetimestamps copied from the timestamped frames of the video contentreceived from the media processor 106. The copied timestamps areexamples of what are more generally referred to herein as “secondtimestamps” that are associated with respective frames of audio contentin the commentator application 115. A given such second timestamp moreparticularly comprises a copy of a corresponding one of the firsttimestamps, with the copy being generated by the timestamp module 122 ofthe commentator application 115. The resulting copied timestamp can thenbe inserted into the appropriate frame of the audio content by thecommentator application 115.

The timestamped audio content is provided by the commentator application115 back to the media processor 106 over network 105, illustrativelyusing a low-speed voice connection but again implemented using aprotocol such as RTMP or WebRTC.

The media processor 106 receives the frames of the audio content fromthe commentator application 115 in association with respective secondtimestamps. For example, the media processor 106 illustratively receivesfrom the commentator application 115 a plurality of audio frames havinginserted therein respective ones of the second timestamps that arecopies of corresponding ones of the first timestamps. The mediaprocessor 106 combines the frames of the video content with the receivedaudio content based at least in part on the first timestamps and thesecond timestamps to generate commented video content that is providedto the CDN web servers 108 for delivery to client devices of respectiveviewers. Such client devices are also referred to herein as “viewerdevices.”

This is an example of an arrangement in which the frames of the videocontent are provided by the media processor 106 in association with thefirst timestamps to the commentator application 115 of the commentatorstation 110-1 at a first relatively low video quality level and thecommented video content is provided by the media processor 106 to theCDN web servers 108 at a second relatively high video quality level.

The term “commented video content” as used herein is intended to bebroadly construed, so as to encompass, for example, a final output videostream, also referred to herein as an “end result” output video stream,or another type of output video stream or other video stream thatincorporates commentary from one or more remote commentators.

The combining of the frames of the video content with the audio contentreceived from the commentator station 110-1 illustratively proceeds inthe following manner in the media processor 106. As mentionedpreviously, the frames of the video content are stored in the mediabuffer 121 of the media processor 106. The media buffer 121, alsodenoted as media buffer M herein, is an example of what is moregenerally referred to herein as a “video frame buffer.”

The frames of the received audio content are processed through atimestamp smoothing module 124 and then stored in an audio frame buffer125, also denoted as commentary buffer C herein. For example, in someembodiments the timestamp smoothing module 124 extracts respective onesof the second timestamps from respective frames of the received audiocontent and applies a smoothing algorithm to the extracted secondtimestamps, such that the smoothed second timestamps can be utilized bythe media processor 106 in combining the frames of the video contentwith the received audio content. Other types of timestamp smoothingarrangements can be used in other embodiments.

The media processor 106 further comprises a delay update module 126,configured to control an updated commentary delay of the received audiocontent. For example, in some embodiments, a current commentary delay Dis determined by the delay update module 126 as a function of a measureddelay of a frame of the audio content and a jitter accommodationcomponent for the commentator application 115. The measured delay of theframe of the audio content in such an arrangement is illustrativelydetermined as a function of a corresponding one of the secondtimestamps, after smoothing, and the processor clock time. The jitteraccommodation component for the commentator application 115 isillustratively determined by processing a plurality of the secondtimestamps, prior to smoothing. Numerous alternative delay updatearrangements can be used.

The video frames from the media buffer 121 are combined with the audioframes from the audio frame buffer 125 by a mixer 127 that includes aninternal media mixer module 128. This combination process illustrativelyutilizes an updated commentary delay provided to the mixer 127 by thedelay update module 126. The resulting commented video content issegmented in an HLS segmenter 129 and delivered from the media processor106 to each of the CDN web servers 108 over the network 105 as shown.

In combining the frames of the video content with the received audiocontent based at least in part on the first timestamps and the secondtimestamps to generate commented video content, the mixer 127illustratively compares a current commentary delay to a designateddelay, and responsive to a difference between the current commentarydelay and the designated delay being above a specified threshold,resetting the designated delay to the current commentary delay. Themixer 127 is further configured to determine a start time of a currentoutput frame of the commented video content as a function of a processorclock time and the designated delay, and to initiate output of thecurrent output frame of the commented video content in accordance withthe determined start time.

The media processor 106 can be further configured to perform additionaloperations in conjunction with combining the frames of the video contentwith the received audio content based at least in part on the firsttimestamps and the second timestamps to generate commented videocontent. For example, the media processor 106 can delete from each ofone or more of the media buffer 121 and the audio frame buffer 125 oneor more frames having timestamps earlier than a current output framestart time. As another example, the media processor 106 can, responsiveto detection of an empty video or audio frame buffer, or a video oraudio frame having a timestamp greater than a sum of the current outputframe time and a frame duration, insert a video or audio lossconcealment frame. Additionally or alternatively, responsive to ameasured delay of an audio frame of the audio content exceeding aspecified threshold, the media processor 106 can signal the commentatorapplication 115 to restart its playback of the video content.

In the FIG. 1 embodiment, the frames of video content are generated bylive media server 102 and its video and surround audio encoder 103 whichprocesses live video from video camera 104 at a live event. The livemedia server 102 is illustratively located at the venue of the liveevent, but other arrangements are possible. There is at least oneregistered remote commentator associated with the commentator station110-1 in a location remote from the live event. The remote commentatorgenerates audio commentary regarding the live event as he or she viewsthe corresponding live video on the commentator station 110-1. The mediaprocessor 106 ensures that the audio and video is correctly synchronizedand mixed before it is segmented and distributed to the CDN web servers108 for delivery to end users.

It should be noted that the live video supplied by the live media server102 to the media processor 106 also illustratively comprises audio, suchas audio encoded with the video utilizing the video and surround audioencoder 103. Such surround audio should be understood to be distinctfrom commentary audio supplied by a remote commentator and mixed withthe live audio in the media processor 106.

As will become more apparent from the embodiments of FIGS. 3 and 4 to bedescribed below, the FIG. 1 embodiment can be extended to accommodatemultiple remote commentators, assuming by way of example that suchremote commentators are located in close enough proximity to one anotherso as to have sufficiently low voice communication delay between them.For example, multiple commentators speaking a particular language canall be located within the same country or geographic region. The contentdelivery system 100 can therefore generate multiple distinct commentedvideo content streams, each with commentary from one or more remotecommentators in a particular language, for delivery to respectivedistinct audiences in different countries or other geographic regions.Accordingly, multiple groups of remote commentators can be present inrespective different countries or other geographic regions with eachsuch group providing audio commentary in the appropriate language forthe content delivered to that country or other geographic region.

Illustrative embodiments allow the remote commentators to work from anylocation with Internet access even if not enough bandwidth is availableto receive the highest original quality video of the event. The end uservideo quality is independent of the quality of the commentator'sdownstream video. Moreover, as asymmetrical connections are still thenorm for home Internet links, the system requires commentators to havesufficient bandwidth to downstream video at some minimal level ofquality, but only up-streams lower bandwidth voice.

Although the remote commentator in some embodiments is at a locationremote from the live event venue, it is possible in other embodimentsfor one or more commentators to be present at the live event venue andadding their commentary to the live video from that site, albeit using aregistered commentator computing station and its associatedfunctionality as illustrated in FIG. 1 instead of conventionalspecialized television studio equipment.

Many different protocols can be used for streaming audio and video inillustrative embodiments. These include the above-noted HLS, MSS, MPEGDASH, RTMP and WebRTC protocols, as well as other protocols such asreal-time transport protocol (RTP). Illustrative embodiments areindependent of the particular video streaming protocol used in any partof the system. That said, a typical embodiment such as that shown inFIG. 1 is illustratively configured to utilize real-time protocols suchas RTMP and WebRTC between the live media server 102, the mediaprocessor 106, the commentator stations such as commentator station110-1, and the distributor. The “distributor” as that term is broadlyused herein is intended to encompass, for example, a contentdistribution entity providing at least a subset of the CDN web servers108. The distributor would typically deliver the content using protocolssuch as HLS, MSS or MPEG DASH that scale well for a large number ofviewers using CDN technology.

In the FIG. 1 embodiment, the content delivery system 100 in its mediaprocessor 106 combines video from a live source with audio-onlycommentary from a remote commentator. The live media serverillustratively compresses the video streamed from a live source, e.g., alive event or an electronic video gaming system. The resulting encodedvideo streams are sent to the media processor 106.

The media processor 106 is shown in FIG. 1 as being implemented in thenetwork 105 but could in other embodiments be co-located with the livemedia server 102 adjacent the network edge. Alternative cloud-basedimplementations of the media processor 106 are also possible. Inaddition, as mentioned previously, there may be multiple instances ofthe media processor 106 distributed at distinct locations within thesystem 100. The media processor 106 in illustrative embodiments isconfigured to synchronize audio from the remote commentator with thelive event video and to forward the mixed stream to the distributor.

To maximize quality for every viewer, the live video uplink from thelive media server 102 to the media processor 106 should have sufficientbandwidth to support high-resolution video streams with highreliability, minimum delay, and low jitter. An important aspect of someembodiments is that the distributor has access to the highest qualityvideo from the original event irrespective of the Internet bandwidthavailable to the remote commentator. For example, illustrativeembodiments allow for a scenario where end users have higher Internetbandwidth than the remote commentator and will receive better videoquality of the original event than that received by the remotecommentator.

Notably, with current streaming protocols, media servers commonly streammultiple versions of the same content, each encoded to different levelsof quality of service (QoS). The viewing device of an end usernegotiates with one or more of the CDN web servers 108 to obtain thebest quality video that its network connection can support. This aspectof the system can be considered standard practice and is therefore notillustrated in FIG. 1.

At the time of ingestion by the media processor 106, each frame of theincoming media stream is timestamped with a corresponding timestamp Tm,by the timestamping module 120 of the media processor 106. Thetimestamped stream is then duplicated with a copy sent to thecommentator station 110-1. Similar to other viewers, the commentatorstation 110-1 receives a stream quality that depends on the supportablebandwidth of the video downlink from the media processor 106 to thecommentator station 110-1. Simultaneously, the video frames are sent tothe media buffer 121, where they are queued until the matchingcommentator's audio stream is received and ready for mixing.

The commentator application 115 installed in the commentator station110-1 allows the commentator to sign on, view, and comment on the event.This can for example be implemented inside a standard browser or as astand-alone software implementation. The commentator application 115will play the video from the original event to the commentator, usuallyvia built-in media players in the browsers or mobile devices, and at thesame time record the commentator's audio.

An important aspect of some embodiments is that the commentatorapplication 115 will copy the timestamps Tm of the original event videoas extracted by the commentator's video player and insert them into theaudio frames being sent back to the media processor 106. This marks theaudio stream with the appropriate timing information from the videoframe, effectively linking what was said with what was seen on-screen bythe commentator at that moment, for later synchronization. The videofrom the original event, however, will not be sent back from thecommentator to the media processor. Thus, the voice uplink from thecommentator station to the media processor can have substantially lowerbandwidth as it is only being used to send back the commentator's audiostream.

The media processor 106 ingests the commentator voice stream and readsthe timestamps Tm from the audio stream before queuing the audio framesin the audio frame buffer 125, which as previously noted is alsoreferred to herein as commentary buffer C. This stream of timestamps Tm,while providing a measure of audio delay, in some cases includes jitterintroduced by a lower quality commentator video downlink as well as thequeuing scheme deployed by the commentator's built-in video player.Therefore, the media processor 106 is configured to apply a smoothingalgorithm in the timestamp smoothing module 124 to the timestamps Tm andthen to re-timestamp the audio stream. For example, a standardexponential smoothing algorithm is applied in some embodiments. Othersmoothing techniques, such as averaging over a sliding window timeperiod, may also be used. The time constant or window size for smoothingis a parameter that can be configured via link quality testing duringinitialization of the commentator's custom application.

Each of the buffers 121 and 125 is assumed to be sized dynamically andcan grow to accommodate ingested data up to preset maximums. Inpractice, to minimize latency, the mixer 127 extracts a frame for mixingshortly after an audio frame and its matching video, appropriatelydelayed by the mixer algorithm, are queued. A standard software-basedmedia mixer module 128 within the mixer 127 can be used to insert thecommentary into the media stream. Finally, the commented media framesare queued and segmented by a standard HLS segmenter 129 fordistribution to the CDN web servers 108.

FIG. 2 shows an example of a mixer algorithm 200 implemented at least inpart by the mixer 127 in the media processor 106 in content deliverysystem 100 of FIG. 1. It is assumed that the mixer algorithm 200 istimer driven to generate video synchronously at a standard frame rateclock interval, e.g., 29.97 Hz.

The mixer algorithm 200 illustratively comprises repeated iterations ofsteps 202 through 226. Although these steps are performed primarily bymixer 127, other modules such as modules 124 and 126, or more generallyother portions of the media processor 106, can perform at least portionsof one or more of the steps.

In step 202, a given iteration of the mixer algorithm 200 is triggeredvia a timer interrupt as shown.

In step 204, the current frame output time range is updated. The currentframe output time range is bounded by frame starting time Tc and Tc+Tf,where Tf is the duration of a video frame. As shown, Tc is set as T−D*,where T is the clock time of the media processor 106. Essentially, thesetting of Tc instructs the mixer 127 to delay the video frame output bya delay D*. D* is periodically reset to the measured current commentarydelay D, when the difference between D and D* exceeds a presetthreshold.

More specifically, D=Td+Tj where Td is the measured current audio delayand Tj is the extra delay added to accommodate potential jitter from thecommentator's voice uplink. At each audio frame's ingestion, Td iscalculated as T−Tm(smoothed). As the overall system is asynchronous, themeasured current commentary delay D is likely to drift over time,necessitating the reset of D*. The reset threshold, however, isconfigured to minimize frequent resets to minimize interruption of theoperation of the media mixer module 128.

Using real-time uplink protocols, Tj should be small and can bedetermined via link quality tests during system initialization. However,the present embodiment allows for Tj to be derived from the Tm databefore smoothing and applied as a dynamic variable.

The mixer 127 then selects the appropriate input video and audio framesto mix from media buffer M and commentary buffer C, respectively, alsodenoted as respective buffers 121 and 125 in FIG. 1. This portion of theprocess is carried out using steps 206 through 222.

In step 206, the mixer 127 checks the timestamp Tm of the oldest videoframe in media buffer M and proceeds as follows.

If Tm<Tc, the oldest video frame is deleted in step 208 and then step206 is repeated. This part of the process therefore removes from mediabuffer M any video frames that are too old, i.e., video frames for whichTm<Tc.

If Tm>Tc+Tf, or media buffer M is empty, the process moves to step 210to insert a loss concealment video frame as the oldest video frame, andthen moves to step 212. This part of the process inserts lossconcealment video frames if queued video frames are out of range ormedia buffer M is empty. Examples of loss concealment video framesinclude the latest complete video frame or an image still.

If Tc<Tm<Tc+Tf, the oldest video frame is in-range and the process movesdirectly from step 206 to step 212 as shown.

In step 212, any audio frames with Tm<Tc are deleted from commentarybuffer C.

At this point, a particular video frame has been identified for mixing,and it remains to identify in step 216, 218 and 220 an appropriatecorresponding audio frame to be mixed with the particular video frame.

In step 216, the mixer 127 checks the timestamp Tm of the oldest audioframe in commentary buffer C and proceeds as follows.

If Tm<Tc, the oldest audio frame is deleted in step 218 and then step216 is repeated. This part of the process therefore removes fromcommentary buffer C any audio frames that are too old, i.e., audioframes for which Tm<Tc.

If Tm>Tc+Tf, or commentary buffer C is empty, the process moves to step220 to insert alternate audio if the empty status has lasted beyond aspecified threshold, and otherwise to insert a loss concealment audioframe, and then moves to step 222. This part of the process insertsalternate audio or loss concealment audio frames if queued audio framesare out of range or commentary buffer C is empty. The alternate audio isused in place of loss concealment audio frames when there are too manyconsecutive missing audio frames to effectively conceal, based upon theabove-noted threshold. The alternate audio illustratively comprises asequence of multiple audio frames, such as an alternate defaultcommentary, prerecorded background music, advertising, generalannouncements or combinations thereof. Examples of loss concealmentaudio frames include low-level white noise or repeated audio.

If Tc<Tm<Tc+Tf, the oldest audio frame is in-range and the process movesdirectly from step 216 to step 222 as shown.

In step 222, the particular identified video frame and the appropriatecorresponding audio frame, which illustratively represent the oldestvideo frame from media buffer M and the oldest audio frame fromcommentary buffer C, are sent to the media mixer module 128 to be mixedtogether to create an output commented video frame.

In step 224, a determination is made as to whether or not the measuredcurrent commentary delay D exceeds a maximum delay tolerance Dx. IfD>Dx, the media processor 106 signals the commentator station 110-1 torestart its video player in order to allow the system to resynchronize,and otherwise does not so signal the commentator station 110-1.

In step 226, the current iteration of the mixer algorithm 200 is exited.Another iteration will be triggered at the next timer interrupt, whichrestarts the process as described above for mixing of the next selectedvideo and audio frames.

It is to be appreciated that the particular process steps of the FIG. 2flow diagram and other flow diagrams herein are presented by way ofillustrative example only, and should not be construed as limiting inany way. Additional or alternative process steps may be used, and theordering of the steps may be varied, in other embodiments. Also, stepsshown as being performed serially in illustrative embodiments can beperformed at least in part in parallel with one another in otherembodiments.

As noted above, some embodiments are configured to support multipleremote commentators. For example, two or more commentators at differentlocations can jointly comment on the same game or other live event forthe same audience. As long as the voice communication delay between thecommentators is within tolerance, illustrative embodiments readilyaccommodate multiple commentators.

In multiple commentator embodiments of this type, a plurality ofdistinct streams of audio content are illustratively received fromrespective distinct commentator applications on respective distinctcommentator stations, and the frames of the video content are combinedwith the plurality of distinct streams of audio content in a mediaprocessor in order to generate the commented video content.

FIG. 3 shows one example of an illustrative embodiment of this type.More particularly, FIG. 3 shows a content delivery system 300 thatcomprises a network 305, a media processor 306 and a plurality ofcommentator stations 310-1, . . . 310-n. It is assumed that the system300 further comprises additional components, such as live media server102, video camera 104 and CDN web servers 108 previously described inconjunction with the FIG. 1 embodiment. The media processor 306generally operates in a manner similar to that previously described formedia processor 106 but its functionality is expanded to accommodatemultiple remote commentators rather than a single remote commentator asin the FIG. 1 embodiment. In addition, a given one of the commentatorstations 310-1, . . . 310-n generally operates in a manner similar tothat previously described for commentator station 110-1.

The commentator stations 310-1, . . . 310-n as illustrated in FIG. 3comprise respective screens 312-1, . . . 312-n, microphones 314-1, . . .314-n, commentator applications 315-1, . . . 315-n, and timestampmodules 322-1, . . . 322-n, which correspond to respective components112, 114, 115 and 122 of commentator station 110-1.

In the FIG. 3 embodiment, the timestamp modules 322-1, . . . 322-n ofthe respective commentator applications 315-1, . . . 315-n of therespective commentator stations 310-1, . . . 310-n copy the timestampsTm from the received frames of the incoming live video streams deliveredby the media processor 306 to their respective audio commentary streamsprior to delivering those audio commentary streams back to the mediaprocessor 306. The media processor 306 comprises a timestamp module 320and a media buffer 321, the latter also denoted as media buffer M, whichcorrespond to respective components 120 and 121 of media processor 106.

The media processor 306 separately smooths and queues the differentaudio commentary streams received from the respective commentatorapplications 315-1, . . . 315-n, utilizing respective timestampsmoothing modules 324-1, . . . 324-n and respective audio frame buffers325-1, . . . 325-n. The audio frame buffers 325-1, . . . 325-n are alsodenoted herein as commentary buffers C(1) through C(n).

The media processor 306 comprises a delay update module 326, configuredto control updated commentary delays for the received audio content fromrespective ones of the commentator stations 310. For example, for thereceived audio content from commentator station 310-n, the delay updatemodule 326 utilizes the smoothed timestamp Tm(n) from timestampsmoothing module 324-n to set link delay D(n) as D(n)=Td(n)+Tj(n) whereTd(n) and Tj(n) are the respective audio delay and jitter tolerance forvoice uplink n from commentator station 310-n to the media processor306. A similar delay update process is performed in delay update module326 for the received audio content from each of the other commentatorstations 310. Although shown as a single module in this embodiment, thedelay update module 326 in other embodiments can be implemented asmultiple separate modules, one for each of the commentator stations fromwhich audio commentary is received.

The media processor 306 further comprises mixer 327 which includes amedia mixer module 328. The mixer 327 is coupled to an HLS segmenter329. These components 327, 328 and 329 correspond generally tocomponents 127, 128 and 129 of the FIG. 1 embodiment. However, the mixer327 of the media processor 306 is configured to synchronize and mix thevideo frames of the live video content with the audio commentary framesreceived from the multiple commentator stations 310. In this embodiment,the video frames are delayed by an amount sufficient to accommodate theremote commentator with the longest delay that is within the maximumdelay tolerance Dx.

FIG. 4 shows an example of a mixer algorithm 400 implemented at least inpart by the mixer 327 in the media processor 306 in content deliverysystem 300 of FIG. 3. It is again assumed that the mixer algorithm 400is timer driven to generate video synchronously at a standard frame rateclock interval, e.g., 29.97 Hz.

The mixer algorithm 400 illustratively comprises repeated iterations ofsteps 402 through 426. Although these steps are performed primarily bymixer 327, other modules such as modules 324 and 326, or more generallyother portions of the media processor 306, can perform at least portionsof one or more of the steps.

The steps 402 through 426 of the mixer algorithm 400 correspondgenerally to respective corresponding steps 202 through 226 of the mixeralgorithm 200 as previously described, but suitably modified toaccommodate the multiple commentator stations 310-1, . . . 310-n.References below to “for all n” should be understood to refer to allindex values from 1 to n, which correspond to respective ones of the nremote commentators. Similarly, certain references to delays D(n),timestamps Tm(n), and commentary buffers C(n) in the figure, as well asreferences to other related parameters in the following description,should be understood from the context to span over all of the indexvalues from 1 to n. Accordingly, in some contexts herein the variable nshould be viewed as an index, encompassing all integer values from 1 ton, and in other contexts it refers only to the particular index value n.The meaning in the various contexts will be readily apparent to thoseskilled in the art.

In step 402, a given iteration of the mixer algorithm 400 is triggeredvia a timer interrupt as shown.

In step 404, the current frame output time range is updated. The currentframe output time range is bounded by frame starting time Tc and Tc+Tf,where Tf is the duration of a video frame. As shown, Tc is set as T-D*,where T is the clock time of the media processor 306. Essentially, thesetting of Tc instructs the mixer 327 to delay the video frame output bya delay D*. D* is periodically reset to the measured current commentarydelay D, when the difference between D and D* exceeds a presetthreshold. In this embodiment, D is set to the largest of the D(n) whichdoes not exceed Dx.

More specifically, D(n)=Td(n)+Tj(n) where Td(n) is the measured currentaudio delay for audio commentator stream n and Tj(n) is the extra delayadded to accommodate potential jitter from the voice uplink ofcommentator n. At each audio frame's ingestion, Td(n) is calculated asT-Tm(n)(smoothed). The remaining delays D(1) through D(n−1) are eachdetermined in a similar manner, and as mentioned previously, themeasured current commentary delay D is set to the largest of the D(n)which does not exceed Dx. As the overall system is asynchronous, themeasured current commentary delay D is likely to drift over time,necessitating the reset of D*. The reset threshold, however, isconfigured to minimize frequent resets to minimize interruption of theoperation of the media mixer module 328.

Using real-time uplink protocols, Tj(n) should be small and can bedetermined via link quality tests during system initialization. However,the present embodiment allows for Tj(n) to be derived from the Tm(n)data before smoothing and applied as a dynamic variable.

The mixer 327 then selects the appropriate input video and audio framesto mix from media buffer M and commentary buffers C(1) through C(n),respectively, also denoted as respective buffers 321 and 325-1, . . .325-n in FIG. 3. This portion of the process is carried out using steps406 through 422.

In step 406, the mixer 327 checks the timestamp Tm of the oldest videoframe in media buffer M and proceeds as follows.

If Tm<Tc, the oldest video frame is deleted in step 408 and then step406 is repeated. This part of the process therefore removes from mediabuffer M any video frames that are too old, i.e., video frames for whichTm<Tc.

If Tm>Tc+Tf, or media buffer M is empty, the process moves to step 410to insert a loss concealment video frame as the oldest video frame, andthen moves to step 412. This part of the process inserts lossconcealment video frames if queued video frames are out of range ormedia buffer M is empty. Examples of loss concealment video framesinclude the latest complete video frame or an image still.

If Tc<Tm<Tc+Tf, the oldest video frame is in-range and the process movesdirectly from step 406 to step 412 as shown.

In step 412, any audio frames with Tm<Tc are deleted from commentarybuffers C(1) though C(n).

At this point, a particular video frame has been identified for mixing,and it remains to identify in step 416, 418 and 420 appropriatecorresponding audio frames from the respective commentary buffers C(1)through C(n) to be mixed with the particular video frame.

In step 416, for all n, the mixer 327 checks the timestamp Tm(n) of theoldest audio frame in each of the commentary buffers C(1) through C(n)and proceeds as follows.

If Tm(n)<Tc, the oldest audio frame is deleted in step 418 and then step416 is repeated. This part of the process therefore removes fromcommentary buffer C(n) any audio frames that are too old, i.e., audioframes for which Tm(n)<Tc.

If Tm(n)>Tc+Tf, or commentary buffer C(n) is empty, the process moves tostep 420 to insert a loss concealment audio frame, and then moves tostep 422. This part of the process inserts loss concealment audio framesif queued audio frames are out of range or commentary buffer C(n) isempty. Examples of loss concealment audio frames include low-level whitenoise or repeated audio. Although not indicated in the figure, it ispossible in other embodiments to use alternate audio in place of lossconcealment audio frames when there are too many consecutive missingaudio frames to effectively conceal.

If Tc<Tm(n)<Tc+Tf, the oldest audio frame of commentary buffer C(n) isin-range and the process moves directly from step 416 to step 422 asshown.

The above-described steps 416, 418 and 420 are performed for each of thedifferent commentator buffers C(1) through C(n) to identify anappropriate corresponding audio frame for each of the n commentators.

In step 422, the particular identified video frame and the appropriatecorresponding audio frames, which illustratively represent the oldestvideo frame from media buffer M and the oldest audio frames fromrespective commentary buffers C(1) through C(n), are sent to the mediamixer module 328 to be mixed together to create an output commentedvideo frame.

In step 424, a determination is made for all n as to whether or not themeasured current commentary delay D(n) exceeds a maximum delay toleranceDx. If D(n)>Dx, the media processor 306 signals the correspondingcommentator station 310-n to restart its video player in order to allowthe system to resynchronize, and otherwise does not so signal thecommentator station 310-n. Also, any buffered audio frames for thecommentator stream with measured current commentary delay D(n)>Dx aredeleted from the corresponding commentary buffer C(n), thereby excludingthose excessively delayed frames from possible selection for mixing withthe video frames.

In step 426, the current iteration of the mixer algorithm 400 is exited.Another iteration will be triggered at the next timer interrupt, whichrestarts the process as described above for mixing of the next selectedvideo and audio frames.

Again, it is to be appreciated that the particular process steps of theFIG. 4 flow diagram and other flow diagrams herein are presented by wayof illustrative example only, and should not be construed as limiting inany way. The process steps can be varied in the manner describedpreviously.

The illustrative embodiments described herein allow, for example, livestreaming of an event with the inclusion of commentary, from one or moreremote commentators who narrate or “call” the event, in a commentedvideo stream that is delivered to end users. As indicated previously,these and other embodiments advantageously avoid the need to sendcommentators to the actual event or to rent a TV production studio andhave the commentators travel to the studio, and can therefore beimplemented at low cost relative to such conventional approaches.Moreover, a wide variety of different types of commentary can bereceived from remote commentators in a given embodiment and incorporatedinto a combined video stream for delivery to end users in accordancewith the techniques disclosed herein. For example, remote commentatorsin some embodiments are enabled to provide audio content, video content,image content, social media posting content, chat text, closed captiontext and/or other types of commentary for inclusion into a commentedvideo stream. These and other embodiments can further allow certainremote commentators to control various aspects of the video productionof the commented video stream that is delivered to end users.

The illustrative embodiments of FIGS. 1 to 4 implement adaptivebuffering techniques that can accommodate relatively large delay andjitter in the connections between the commentator stations and the mediaprocessor. However, long link delays can negatively impact the abilityof multiple commentators to interact effectively with one another, assuch interaction in some embodiments can require that the delays betweenthe commentators be below about 100 milliseconds. Nevertheless,commentators with long link delays can be accommodated if they areproviding independent commentary. One such scenario would be to have acommentator posting voice, another posting text, and another doing signlanguage picture-in-picture (PIP) video over the main broadcast stream.

In embodiments that require effective interactions between commentators,the commentators can be configured to use low-latency RTMP or WebRTClinks to a cloud-based media processor. With such links, the adaptivebuffering techniques described above can be greatly simplified. Forexample, buffer length can be set once for a given session and need notbe adjusted throughout the session. However, to reduce re-encodingoverhead, which will add delay when commentators interact with oneanother, and to provide other significant advantages, alternativetechniques utilizing other types of mixer arrangements can be deployed,as will be described in more detail below.

Some embodiments therefore need not utilize adaptive bufferingtechniques that attempt to minimize overall delay to the extentpossible. These embodiments are instead particularly configured toprovide quality interactivity between commentators, while tolerating agreater amount of overall delay.

Such arrangements provide stricter limits on the real-time nature of thecommentator's links with respect to each other and the low qualityversion of the video stream. The overall delay to the greater viewerpublic, however, can be substantial. For example, if the live videosource is in the U.S. while the target audiences are in China, thecommentators could all be in China. The relatively low-quality versionof the live video that is commented on by each of the commentators canbe delayed, possibly by as much as a couple of seconds, before beingviewed by the commentators. However, in some embodiments of this type,all of the commentators have low-latency RTMP or WebRTC links to cloudservers in China. As a result, the commentators have low latencyrelative to one another and can interact with one another and with thedelayed low-quality video with ease. The high-quality version of thebroadcast video could be delayed even further, in order to providesufficient time for synchronization and mixing of all of the commentatorinput streams. Again, adaptive buffering techniques of the typedescribed in conjunction with FIGS. 1-4 are not needed.

Additional illustrative embodiments involving remote commentary will nowbe described with reference to FIGS. 5, 6 and 7. These additionalembodiments are respective examples of cloud-based commentator systemswhich provide advantages similar to those described above in the contextof other embodiments. For example, the cloud-based commentator systemsin these embodiments do not require a traditional TV studio or otherequipment, and instead provide a solution that allows each of a set ofcommentators to be at “home” (e.g., on a consumer Internet connection,with a consumer-level desktop computer, laptop or other computingdevice). Moreover, the cloud-based commentator systems in theseembodiments enable each commentator to provide a wide variety ofcommentary, including one or more of audio content, video content, imagecontent, social media posting content, chat text and closed captiontext, and further allow one or more of the commentators to controlvarious aspects of the video production of an end result output videostream provided to end users.

The cloud-based commentary systems in these embodiments can be used in awide variety of different remote commentator scenarios, as will becomemore apparent from the following description. For example, in someembodiments, commentators are enabled to add both audio and video to anend result output video stream of an event. The audio (e.g., the voiceof the commentator) is mixed in with the audio of the event, and videoof or provided by the commentator is mixed in with the video of theevent, such as using various PIP video arrangements, overlays, etc.Other types of media content, such as images and/or chat text, can beused in addition to or in place of audio and video.

A given such cloud-based commentator system in some embodiments isillustratively configured to enable multiple commentators each in his orher own location. Such multiple commentator arrangements may be for agroup commentator session for a single event that generates one outputvideo stream, or can be for a single event commentated on by multiplecommentators generating separate or multiple output video streams. Themultiple commentators, in some cases, may be geographically dispersed.For example, commentators may be located in multiple continents or othergeographic regions. Further, it is assumed that each of the commentatorshas at least a consumer-level personal desktop computer, laptop or othercomputing device with a standard Internet browser installed. Thecommentators are not required to have any specialized software installedon their computing devices, other than a commentator application of thetype disclosed herein that is configured to support the remotecommentator functionality. Such a commentator application can beimplemented, for example, as a component of an otherwise conventionalweb browser. However, it is to be appreciated that web browsers are notrequired, and can be replaced in other embodiments by a wide variety ofother types of application programs, such as native desktop applicationsor other computer applications that do not operate as or otherwiseinclude web browsers.

In some embodiments, a live event is assumed to have multiple availableinput video streams associated therewith. The multiple input videostreams may be obtained from respective ones of multiple camera streamsgenerated at the event, although in other embodiments the multiple inputvideo streams can comprise individual streams from multiple events, etc.One or more of the commentators is illustratively configured to controlhow the input video streams are combined into a final output videostream.

The cloud-based commentator system is also configured to provide orenable the use of graphic add-ons to the input video streams of the liveevent. Such graphic add-ons may include, but are not limited to, images,slates (e.g., a video slate used to facilitate video production), videoclips, and social media postings (e.g., tweets from Twitter), each anexample of “commentary information” as that term is broadly used herein.Each of the commentators is illustratively configured to control howsuch graphic add-ons or elements are added to the final output videostream. Alternatively, only a designated subset of the commentators maybe authorized to perform such functions.

In some embodiments, there are multiple destinations for the finaloutput video stream or streams. Such multiple destinations include, butare not limited to, traditional TV, social media platforms (e.g.,Facebook, Twitter, YouTube, etc.), a variety of web sites and mobileapplications, etc.

Referring now to FIG. 5, a cloud-based commentator system 500 is shown,illustrating the flow of streams between a live event 501, cloud 505, aset of commentators 510-1, 510-2 and 510-3, and multiple viewersillustratively comprising YouTube end users 516, Facebook end users 517,and TV end users 518. The cloud 505 illustratively implements one ormore media processors each comprising components such as mixers andservers, as described in more detail elsewhere herein. The commentators510 illustratively comprise respective commentator devices, such ascomputers or mobile devices, each corresponding to a different remoteuser and implementing a commentator application. The term “commentator”in the following description is generally used to refer to thecommentator device, but may additionally or alternatively refer to itscorresponding remote user.

Although FIG. 5 shows an example system in which there are threecommentators, this is for purposes of illustration only. The number ofcommentators may vary as desired for a particular live event. Further,the number of streams of the live event may vary as desired (e.g., moreor less than three as illustrated). Still further, the number and typeof end users or other viewers of the live event may also vary.

The video of a commentator, in some embodiments, is added through a PIPlayout. When there are several commentators, as in the case ofcommentators 510, one or more designated layouts are used to control therelative positioning between the original event video and the PIPs ofeach of the commentators. Note that these layouts may need to changedynamically during a game or other live event. For example, the layoutof the original event video and the commentator PIPs in some embodimentschanges dynamically as commentators 510 join or leave during the liveevent.

It is important in some embodiments to have what is referred to hereinas “tight” synchronization between the event stream and each of thecommentators, as will be described in more detail below. With such tightsynchronization, the end users will perceive the event stream as well asthe voice and PIP of each of the commentators as happening at the sametime. More particularly, the event streams and the commentators 510 aretightly synchronized in some embodiments, by ensuring that there is nomore than a specified maximum amount of delay, typically around 50-100milliseconds, between them. Such an arrangement advantageously allowsthe commentators 510 to have a live interaction with each other and withthe event stream, where each commentator is able to see and hear theevent as well as the commentary from the other commentators.

Accordingly, in the cloud-based commentator system 500 of FIG. 5, it isassumed that the connections between the commentators 510 and componentsof the cloud 505 are bidirectional and tightly synchronized (e.g., withdelays between 50-100 milliseconds).

In some embodiments, it is possible to allow delay between the originallive event 501 and certain components of the cloud-based commentatorsystem 500 as well as between the cloud-based commentator system 500 andthe distribution platforms which supply the end users 516, 517 and 518.For example, it is typically acceptable for the end users 516, 517 and518 to watch the event several seconds after the commentators 510 addedtheir comments, as there is no tight synchronization needed between theend users 516, 517 and 518 and the commentators 510. Given that thecommentators 510 in some cases are remote, it is also acceptable for thecommentators 510 to add their comments a few seconds after the actuallive event as long as their comments are tightly synchronized with theevent stream and with each other.

The commentators 510 can add additional multimedia data to the eventsuch as replays, images, tweets, statistics, etc. These are examples ofwhat are more generally referred to herein as “commentary information.”Other examples include audio, video, social media posts, and closedcaption text.

In some embodiments, the commentators 510 can dynamically change thelayout which specified how the input streams and the PIPs are mixedtogether in the cloud 505. These changes generally need to besynchronized but a delay of around one second is acceptable in someembodiments for a layout or content change. However, these changes arevisible to all commentators at substantially the exact same time, inaccordance with the tight synchronization requirement described above.

The production of the final output stream in cloud 505 to be sent to theend user platforms can also have an additional delay of a few seconds.This delay can be used to make sure that all media elements aresynchronized even though they came through different paths. The delaycan also be used to make sure that a corresponding video encoderachieves the best quality per bandwidth ratio for various bandwidths.

The end users 516, 517 and 518 typically watch the combined event (e.g.,the final output stream) as a stream on a digital device (e.g., alaptop, tablet, or smartphone) or as a broadcast on a traditional TVset. These devices often have high definition and large resolutionscreens. Therefore, it is important to deliver the highest possiblevideo quality to the end users. End users hence are lessdelay-sensitive, but more quality-sensitive.

Video encoders have a trade-off between delay, bandwidth, and quality.It is possible to get the best quality for a certain bandwidth at thecost of a higher delay. Alternatively, it is possible to get tightsynchronization and small delays at the expense of bandwidth andquality. There are typically two types of systems used in the industry:live streaming systems and video conference systems. Live streamingsystems have a higher delay (e.g., multiple seconds) with a good qualityto bandwidth ratio. Video conference systems have lower delays (e.g.,50-100 milliseconds) with a lesser quality to bandwidth ratio.

The above example remote commentator scenario cannot be implementedusing traditional streaming systems between the parties as the higherdelays would make tight synchronization between the commentators and theevent impossible. The above scenario can also not be accomplished with atraditional video conference system, as this would require much higherbandwidth than is typically available on a consumer Internet connectionand much higher processing power than is typically available onconsumer-level laptops or other computing devices. Hence, if one usestraditional video conference system technology with consumer Internetand personal computers, the resulting quality of the video would be verypoor. Alternatively, one could get very high quality video with atraditional video conference system, but it would require dedicatedbandwidth connections and high end compute servers. This would beprohibitively expensive and therefore not suitable for use in thescenario described above where the commentators are at home.

In some embodiments, the cloud-based commentator system 500 provides ahybrid system that uses both video conferencing and streamingtechnologies to meet the requirements outlined above. Further, thecloud-based commentator system 500 allows commentators 510 to add a widevariety of commentary to a live event. Commentators 510 are configuredto add not only their voice, but also video to the event. Thecommentators 510 are also configured to affect the production and layoutof the live event. For example, the commentators can switch inputstreams, add clips, tweets, images, slates and other commentary tofurther customize the output video stream provided to the end users.

The cloud-based commentator system 500, in some embodiments, utilizesone or more mixers and a media server. The media server is part of thecloud 505 and provides the central component responsible for routingvideo between each of the commentators 510 and the mixers. The mixersillustratively comprise a pre-mixer used to generate a relatively lowresolution version of the event stream that is received by thecommentators 510 via the media server, a plurality of instances of acommentary mixer associated with respective ones of the commentators510, and a post-mixer configured to generate the final output videostream utilizing a relatively high resolution version of the eventstream and the commentary information received from the commentators 510via the media server. The post-mixer in some embodiments is configuredto combine different multimedia elements such as video streams, images,and graphics into one output video stream following a prescribed layout.The pre-mixer and the post-mixer, like the media server, are assumed tobe implemented in the cloud 505 of system 500, while the instances ofthe commentary mixer are implemented in respective ones of thecommentator devices of the commentators 510.

More detailed examples of the above-described mixer and media servercomponents will now be presented in conjunction with the illustrativeembodiments of FIGS. 6 and 7. It is to be appreciated that these areonly examples, and other arrangements of mixers, media servers or othermedia processor components can be used.

FIG. 6 shows a cloud-based commentator system 600 that processes eventstreams from a live event 601 using a pre-mixer 602, a media server 603,a post-mixer 604 and a plurality of instances of a commentary mixer onrespective ones of commentators 610-1, 610-2 and 610-3. The pre-mixer602, media server 603 and post-mixer 604 are illustratively implementedin a cloud, similar to cloud 505 of system 500. A final output videostream generated by the post-mixer 604 is made accessible to end users616, 617 and 618 associated with respective distinct distributionplatforms, illustratively YouTube, Facebook and TV, as in the FIG. 5embodiment, although other distribution platforms can be used.

In the FIG. 6 embodiment, three different types of mixers are used,namely, the pre-mixer 602, a commentary mixer comprising mixer instanceson the respective commentators 610, and the post-mixer 604. These threedifferent types of mixers are also denoted as respective Mixer A, MixerB and Mixer C in the figure, with Mixer B comprising the multipleinstances of commentary mixers on the respective commentators 610.Mixers A and C are illustratively implemented as cloud-based mixers inthis embodiment, and Mixer B is illustratively implemented as multiplebrowser-based commentary mixer instances deployed in respective webbrowsers of the commentators 610.

It is to be appreciated that the term “web browser” as used herein isintended to be broadly construed, so as to encompass, for example,numerous types of software programs for use in interacting with webservers. A web browser can include a commentator application as acomponent of the web browser, or the web browser can be implemented as acomponent of a commentator application.

Numerous other arrangements are possible, such as ones in which the webbrowser and the commentator application are entirely separate from oneanother, but interact with one another in supporting the disclosedfunctionality.

Again, as indicated previously herein, web browsers are not required,and a wide variety of alternative application programs, not comprisingor otherwise associated with web browsers, can be used.

Mixer A merges the various inputs from the live event 601 into onestream that is sent to the commentators 610 via the media server 603.Mixer A, in some embodiments, creates a lower-quality stream to be sentto the commentators 610 (e.g., over consumer Internet connections).Mixer A is illustratively configured to have no more than a designatedmaximum delay, which may be around one second.

Mixer B (with its respective instances denoted as “B” in the circlesrepresenting commentators 610 in FIG. 6) is a browser-based commentarymixer. Mixer B is configured to have a delay less than some designatedthreshold (e.g., a delay of a less than 50 milliseconds). As illustratedin FIG. 6, there is one instance of Mixer B per commentator. The variousinstances of Mixer B are assumed to run in a client web browser used byeach of the commentators, and can have lower quality but with lowerdelay (e.g., relative to Mixer A). Through the media server 603, each ofthe commentators 610 receives a stream from the live event 601 as wellas a stream from each other commentator. The local client of eachcommentator (e.g., a web browser) then decodes each stream and combinesthe video and audio for the commentator according to the layout. Nore-encoding is needed, but the streams need to be tightly synchronized(e.g., within 50 milliseconds). In some embodiments, all commentatoractions happen within the web browser and there is no need to installany specialized software on the commentator device, other than thecommentator application that provides the Mixer B instance in support ofthe commentator functionality disclosed herein.

Mixer C merges the commentary streams received from the commentator 610via the media server 603 with one or more event streams from the event601 to provide a final output video stream for delivery to the end users616, 617 and 618. Mixer C can have multiple seconds of delay. Mixer Ccreates one standard output stream for the end users to be deliveredthrough CDN/Internet technology. There is only one instance of Mixer C,which can take several seconds to provide its associated mixing but mayhave a much higher quality to bandwidth ratio (e.g., relative to Mixer Aand Mixer B). The total delay of Mixer C is a combination of thedecoding time of each of the commentator streams, the delay to achieveclose synchronization, the rendering of the combined video, images, andaudio, and the re-encoding at high quality.

Close synchronization of the multiple commentator streams with the eventstream is achieved in some embodiments utilizing techniques similar tothose described above in conjunction with the illustrative embodimentsof FIGS. 1 through 4, although other synchronization techniques can beused in other embodiments.

The interworking between the different types of mixers, and their variedsensitivity to delay, enables the cloud-based commentator system 600described herein to provide the desired functionality. In someembodiments, Mixer A has a maximum one second delay that is acceptablefor input stream switching, Mixer B has a low delay (e.g., 50 to 100milliseconds) required for real time interaction of the commentatorswith the content and with one another, and Mixer C has a multiple seconddelay to get the best quality per bandwidth for the final output videostream. The use of these different mixer types in an architecture for acloud-based commentator system advantageously facilitates theachievement of the desired features and functionality for multipleremote commentators.

For example, a mixer such as Mixer C could generally not be used for allmixing in the system 600, as the delay would not allow for liveinteractions between the commentators. It is also not effective to use amixer such as Mixer B for all interactions, as it would require eithermuch higher Internet speeds or much higher processing capabilities atthe commentators, which would make the solution prohibitively expensive.Furthermore, a mixer such as Mixer B generally cannot be used by the endusers as this would require special adaptation of the client for digitalstreaming and would simply be impossible for the regular TV end users.It is also not possible for a mixer such as Mixer B to perform all ofthe tasks of Mixer A, as a consumer level laptop or other similarcomputing device does not have enough processing power for such tasks.Also, the commentator's Internet connections might not be sufficient tobring all these streams to each commentator.

It should be noted that Mixer C should always get the original inputstreams at the highest quality from the event 601. Thus, while it ispossible for Mixer A to downgrade the quality of the input streams tosuit the Internet connections of the commentators, this will neveraffect the video quality of the input streams going to Mixer C. Forexample, a commentator on a low speed Internet connection could commenton the game while watching it in standard definition (SD), while theactual end users will see the event video with the synchronized commentsin high definition (HD). Also, if a single commentator's Internetconnection were to fail, it will only impact that commentator's voiceand PIP. All other elements will be unaffected.

For the cloud-based commentator system 600 to function as desired,Mixers A, B and C need to remain synchronized. This means that Mixers A,B and C make the same changes in layout and media content at the sametime. This makes it possible for the commentators to make productionchanges to the media content and layout structure, which are thenproperly propagated to all the other commentators and to the end users.Each commentator can affect the layout for themselves and all the othercommentators, as well as the end users.

An example use case is where one of the commentators sees somethinginteresting in one of the input streams, and wants to draw attention toit by switching to that stream and making it full screen. The layout ofall the commentators and end users will follow. Production changesinclude, but are not limited to, adding or removing a new input stream,adding or removing a commentator, switching commentator PIPs on or off,changing how multiple input streams and multiple PIPs are laid out in amixer, adding images or slates to either obscure the video or to serveas a background, adding video clips which can be played as part of thelive stream, rendering tweets or other social media postings, etc.

The synchronization of Mixers A, B and C, in some embodiments, isachieved through the use of a predefined layout language and a shareddata structure in the cloud-based commentator system, as will now bedescribed in more detail.

FIG. 7 shows a cloud-based commentator system 700 that processes eventstreams from a live event 701, using a pre-mixer 702, a media server703, a post-mixer 704 and a plurality of instances of a commentary mixeron respective ones of multiple commentators 710, to generate a finaloutput video stream for end users 716, 717 and 718. These systemcomponents are generally configured to operate in a manner similar tothat of the corresponding components of system 600 as previouslydescribed. However, in the system 700, each of the pre-mixer 702, thecommentator mixer instances on commentators 710, and the post-mixer 704,interacts with a layout data store 720 as shown.

The layout data store 720 is illustratively configured to facilitatesynchronization of Mixers A, B and C and the media server 703. Theabove-noted predefined layout language may be comprise Hypertext MarkupLanguage (HTML), although other languages such as JavaScript can be usedin other embodiments. Each layout prescribes which input, PIP streams,and media elements are combined and how they are laid out with respectto each other. The layouts may include, but are not limited to, oneevent input stream with one commentator PIP rendered on top, one inputstream with four commentators rendered to the left of the input stream,four input streams rendered each in a quadrant of the screen (e.g., a“quad split” layout), etc. In some embodiments, commentators are enabledto predefine any desired number of layouts, and then dynamically switchbetween them during a live event.

The mixers in the cloud-based commentator system 700 utilize differentparts of the layout definition for which they are responsible. Mixer A,in some embodiments, only uses the part of the layout definition thataffects how the input streams and graphics are mixed together. Mixer Btakes the output of Mixer A and adds the PIPs of the commentators in thebrowser of each commentator according to a PIP part of the layoutdefinition. Mixer C will take all inputs from the event, the media, andthe PIPs or other commentary and mixes them together using the fulldefinition of the layout.

As indicated above, the layout data store 720 is used to facilitate thissynchronization of Mixers A, B and C in system 700. The layout datastore 720 is illustratively implemented in the form of a sharedin-memory cloud-based data structure. For example, in some embodiments,the layout data store is implemented using cloud storage technology suchas the Firebase system provided by Google.

In some embodiments, a synchronization algorithm proceeds as follows.Assume that one of the commentators 710 wants to make a change in themix layout, or wants to add multimedia elements such as images, tweets,video clips or other commentary to the output stream. The commentatorenters the desired layout changes in their client (e.g., such as througha graphical user interface (GUI)). A web client of the commentator sendsthe changes to the layout data store 720, and the layout data store 720alerts the other commentator instances of Mixer B, as well as thecloud-based Mixers A and C, of the updates to the layout for the outputstream. Each of the Mixers A, B and C then executes the updated layoutat the prescribed time.

As indicated previously, it is possible in some embodiments for thecommentators using a cloud-based commentator system to be geographicallydispersed, such as on different continents, leading to increases in thepropagation delays between the commentators. The typical protocol usedby a media service (e.g., WebRTC) to communicate with all thecommentators may not work well if the commentators are physically farapart (e.g., geographically dispersed on multiple continents). As aresult, the synchronization may be lost or the video quality maydegrade. Such issues are addressed in illustrative embodiments byextending the cloud-based commentator system with additional mediaservers. For example, media servers may be located in each continentwhere commentators are expected to be located. Each media server willconnect with all the commentators in a particular region (e.g., on itsassociated continent) as well as with Mixers A and C. This can be doneusing a media server protocol such as WebRTC. The various media serversalso communicate with each other, but using a different protocol thatworks better over long distances, such as Secure Reliable Transport(SRT).

When multiple media servers are used, a commentator will find theclosest media server and send its stream there. The closest mediaserver, in some embodiments, is determined algorithmically byempirically testing metrics such as Internet throughput, latency,routing architecture, etc.

It should be appreciated that multiple media servers may also be used toadd redundancy and fail-over to a cloud-based commentator system.

In some embodiments, it is assumed that all commentators have equalpermissions. In other words, each of the commentators is assumed to havefull control over layout and production changes in the output videostream. When there are many commentators, or in cases where somecommentators are less qualified to use the cloud-based commentatorsystem, it is possible to have different permission models for differentones of the commentators. For example, some commentators would only beallowed to add their voice and PIP, but would not have any permissionfor layout or production changes. One commentator, or a fewcommentators, could then play the role of the overall productionmanager, communicating with each of the commentators and centralizingthe production and layout changes.

The input streams from an event in some embodiments come with differentdelays. The different delays may result from various factors. Forexample, some cameras providing the input streams may be traditional TVcameras that go through a TV production chain at the event while othercameras (e.g., a GoPro camera mounted on an athlete's helmet) go througha pure digital transport model. In an e-sport setting, some inputstreams are from cameras pointed at the gamers, while other inputstreams are screen captures of each of the e-game consoles. In these andother scenarios, the input streams arriving in a production cloud suchas cloud 505 may not be synchronized with one another. Thus, someembodiments add delays at the cloud ingestion point to make sure theinput streams each get an individualized additional delay to fullysynchronize them. This may be done manually, using synchronizationmarkers that provide a marked event in all streams, suitable for use ina synchronization process. Examples of such synchronization markersinclude markers resulting from use of a movie “clapper,” or markersresulting from inserting an image of a synchronized clock on all theinput streams. It is also possible to implement algorithms to detectsuch synchronization markers and to automatically adjust the delays forthe input streams.

In some of the above-described embodiments, the commentators interactwith each other and with the low resolution live-event stream in ateleconferencing session via Mixer B. While a wide variety of otherteleconferencing systems can be adapted to this purpose in the mannerdisclosed herein, the use of a browser-based system as described aboveis particularly cost-effective. Moreover, no transcoding is necessary.Each commentator's browser simply shows the other commentators' outputstreams in separate windows. In other words, the mixing is readilyapparent on each commentator's screen. Moreover, as long as the delaysfrom the commentators are within certain limits, each commentator willnaturally compensate for the small delays from the other commentators.As indicated above, sufficient delay is added for the high resolutionlive-event stream to accommodate all the commentators' delays, includingthe additional delay to get to Mixer C.

It should once again be noted that a wide variety of different types ofcommentary can be provided, including by way of example audio, video,images, social media posts and closed caption text.

In some embodiments, each commentator can add live closed caption textinto his or her commentator stream. For example, a human commentatoradds (e.g., types) in live closed caption text while the event ishappening. Such an arrangement can use the same synchronizationtechniques described elsewhere herein for commentator audio, butpossibly with an added feature to “speed up” the typed comments a bit inorder to adjust for the time it takes to type them, thereby improvingthe synchronization. It is also possible for the commentator to use aspeech-to-text conversion system to automatically generate closedcaption text from audio of commentator speech. Such a speech-to-textsystem can be integrated into the overall architecture of the system,possibly as a cloud-based component.

In other embodiments, one or more of the commentators can add video thatincludes sign language commentary. For example, a commentator could adda PIP showing that commentator translating the event commentary intovisual sign language.

As yet another example, the commentators can interact via a “chat”arrangement that utilizes text rather than audio. Such text entered bythe commentators using their respective commentator applications isintended to encompass various types of messaging text and is moregenerally referred to herein as “chat text.”

Again, these are only examples of various types of commentaryinformation that can be inserted into a final output video stream usingthe techniques disclosed herein. Such commentary information can includea wide variety of different media types, as well as combinations ofmultiple different media types.

As is apparent from the foregoing, illustrative embodiments disclosedherein are readily scalable to potentially large numbers ofcommentators, at least some of which are present at respective locationsthat are remote from a live video event venue.

Moreover, illustrative embodiments disclosed herein can be implementedat least in part using standard based built-in clients and HTTP servers,and thus at substantially reduced cost and complexity relative toconventional approaches.

Illustrative embodiments are not limited to use with the HLS protocol.For example, the disclosed embodiments can be adapted to save bandwidthwith any HTTP based streaming protocol, including the MSS and MPEG DASHprotocols. Moreover, it is to be appreciated that other embodiments canbe configured utilizing a wide variety of other types of streamingprotocols and accordingly are not limited to use with live streaming orHTTP.

Embodiments of the invention can be implemented using any type of mobiledevice or more generally any other type of client device, including, forexample, desktop, laptop or tablet personal computers, smarttelevisions, smart watches, gaming systems and other processing devices.

It should once again be noted that the above-described arrangements areexemplary only, and alternative arrangements can be used in otherembodiments.

The disclosed techniques can also provide significant advantages innumerous content delivery contexts other than live event video.

A given client, server or other component in a content delivery systemas disclosed herein is configured utilizing a corresponding processingdevice comprising a processor coupled to a memory. The processorexecutes software code stored in the memory in order to control theperformance of processing operations and other functionality. Theprocessing device also comprises a network interface that supportscommunication over one or more networks.

The processor may comprise, for example, a microprocessor, anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA), a central processing unit (CPU), an arithmetic logicunit (ALU), a digital signal processor (DSP), a graphics processing unit(GPU) or other similar processing device component, as well as othertypes and arrangements of processing circuitry, in any combination.

The memory stores software code for execution by the processor inimplementing portions of the functionality of the processing device. Agiven such memory that stores software code for execution by acorresponding processor is an example of what is more generally referredto herein as a computer-readable storage medium having computer programcode embodied therein, and may comprise, for example, electronic memorysuch as SRAM, DRAM or other types of random access memory, read-onlymemory (ROM), flash memory, magnetic memory, optical memory, or othertypes of storage devices in any combination.

Articles of manufacture comprising such computer-readable storage mediaare considered embodiments of the invention. The term “article ofmanufacture” as used herein should be understood to exclude transitory,propagating signals.

In addition, embodiments of the invention may be implemented in the formof integrated circuits comprising processing circuitry configured toimplement processing operations, such as mixing of live video andaudio-only commentary from multiple remote commentators, associated withcontent delivery.

The particular configurations of content delivery systems describedherein are exemplary only, and a given such system in other embodimentsmay include other elements in addition to or in place of thosespecifically shown, including one or more elements of a type commonlyfound in a conventional implementation of such a system.

For example, in some embodiments, a content delivery system may beconfigured to utilize the disclosed techniques to provide additional oralternative functionality in other contexts. The disclosed techniquescan be similarly adapted for use in a wide variety of other types ofcontent delivery systems.

It is also to be appreciated that the particular process steps used inthe embodiments described above are exemplary only, and otherembodiments can utilize different types and arrangements of processingoperations.

It should again be emphasized that the embodiments of the invention asdescribed herein are intended to be illustrative only. Other embodimentsof the invention can be implemented utilizing a wide variety ofdifferent types and arrangements of content delivery systems, networks,and devices than those utilized in the particular embodiments describedherein. In addition, the particular assumptions made herein in thecontext of describing certain embodiments need not apply in otherembodiments. These and numerous other alternative embodiments will bereadily apparent to those skilled in the art.

What is claimed is:
 1. A method comprising: receiving from each of aplurality of commentator applications corresponding commentaryinformation relating to video content from at least one video source;sending at least portions of the commentary information received fromone or more of the commentator applications to one or more other ones ofthe commentator applications; and generating commented video contentbased at least in part on the commentary information received from thecommentator applications; wherein receiving from each of the commentatorapplications corresponding commentary information relating to the videocontent comprises receiving a plurality of distinct streams of mediacontent from respective ones of the commentator applications; whereinthe commented video content is provided to one or more servers of acontent delivery network for delivery to one or more viewer devices; andwherein the method is performed by at least one processing devicecomprising a processor coupled to a memory.
 2. The method of claim 1wherein the video content comprises live video from at least one livevideo source.
 3. The method of claim 1 wherein the stream of mediacontent received from a corresponding one of the commentatorapplications comprises at least one of audio content, video content,image content, social media posting content, chat text and closedcaption text.
 4. The method of claim 1 wherein the commentatorapplications receive respective instances of a designated version of thevideo content from said at least one video source.
 5. The method ofclaim 4 wherein the designated version of the video content is generatedin a pre-mixer of said at least one processing device at no more than afirst resolution, utilizing a plurality of content streams fromrespective ones of a plurality of video sources, with each of theplurality of content streams having at least a second resolution wherethe second resolution is higher than the first resolution.
 6. The methodof claim 1 wherein the commentator applications are implemented onrespective commentator devices that are separate from said at least oneprocessing device that performs the method.
 7. The method of claim 6wherein the receiving and sending are implemented in a media server ofsaid at least one processing device through interaction of the mediaserver with respective web browsers of the respective commentatordevices.
 8. The method of claim 7 wherein each of the web browsersimplements an instance of a commentary mixer configured to combinecommentary information from its corresponding commentator applicationwith additional commentary information received from respective otherones of the commentator applications via the media server.
 9. The methodof claim 8 wherein the instances of the commentary mixers implemented byrespective ones of the web browsers are synchronized with one anotherrelative to the video content to less than a specified amount of delayin order to support apparent real-time interaction between users of thecommentator applications in the commented video content as viewed at theone or more viewer devices.
 10. The method of claim 7 wherein a firstweb browser of a first one of the commentator devices is configured topresent commentary information received from respective other webbrowsers of other ones of the commentator devices via the media server,wherein the commentary information received from the respective otherweb browsers via the media server is presented by the first web browserin respective distinct display windows.
 11. The method of claim 6wherein generating the commented video content comprises generating thecommented video content by combining, in a post-mixer of said at leastone processing device, at least portions of the commentary informationreceived from the commentator applications with a designated version ofthe video content from said at least one video source.
 12. The method ofclaim 1 wherein at least a first one of the commentator applications isconfigured to permit incorporation into the commentator information atleast one of audio content, video content, image content, social mediaposting content, chat text and closed caption text, and to permitcontrol of a manner in which such commentator information is to beutilized in generating the commented video content.
 13. The method ofclaim 1 wherein generating the commented video content based at least inpart on the commentary information received from the commentatorapplications comprises: storing a plurality of distinct layouts eachspecifying a different arrangement of the commentator information in thecommented video content; and dynamically switching between at least asubset of the distinct layouts over time in generating the commentedvideo content.
 14. The method of claim 13 wherein a given one of thedistinct layouts specifies for each of one or more of the commentatorapplications at least a positioning of corresponding commentatorinformation in the commented video content.
 15. The method of claim 13wherein the distinct layouts are stored in a layout data store that isaccessible to each of: a pre-mixer used to generate a first resolutionversion of the video content received by the commentator applications; aplurality of instances of a commentary mixer associated with respectiveones of the commentator applications; and a post-mixer configured togenerate the commented video content utilizing a second resolutionversion of the video content and the commentary information receivedfrom the commentator applications; wherein the second resolution versionhas a higher resolution than the first resolution version.
 16. Themethod of claim 1 wherein the commentary information received from agiven one of the commentator applications comprises additionalmultimedia information not part of the video content from the at leastone video source.
 17. The method of claim 16 wherein the additionalmultimedia information comprises at least one of an image, a video and asocial media post.
 18. The method of claim 16 wherein the additionalmultimedia information comprises at least one replay of at least aportion of the video content from the at least one video source.
 19. Anarticle of manufacture comprising a non-transitory computer-readablestorage medium having computer program code embodied therein, whereinthe computer program code when executed in at least one processingdevice causes said at least one processing device: to receive from eachof a plurality of commentator applications corresponding commentaryinformation relating to video content from at least one video source; tosend at least portions of the commentary information received from oneor more of the commentator applications to one or more other ones of thecommentator applications; and to generate commented video content basedat least in part on the commentary information received from thecommentator applications; wherein receiving from each of the commentatorapplications corresponding commentary information relating to the videocontent comprises receiving a plurality of distinct streams of mediacontent from respective ones of the commentator applications; andwherein the commented video content is provided to one or more serversof a content delivery network for delivery to one or more viewerdevices.
 20. The article of manufacture of claim 19 wherein thereceiving and sending are implemented in a media server throughinteraction of the media server with web browsers of respectivecommentator devices that implement the commentator applications.
 21. Thearticle of manufacture of claim 20 wherein each of the web browsersimplements an instance of a commentary mixer configured to combinecommentary information from its corresponding commentator applicationwith additional commentary information received from respective otherones of the commentator applications via the media server.
 22. Anapparatus comprising: at least one processing device comprising aprocessor coupled to a memory; wherein said at least one processingdevice is configured: to receive from each of a plurality of commentatorapplications corresponding commentary information relating to videocontent from at least one video source; to send at least portions of thecommentary information received from one or more of the commentatorapplications to one or more other ones of the commentator applications;and to generate commented video content based at least in part on thecommentary information received from the commentator applications;wherein receiving from each of the commentator applicationscorresponding commentary information relating to the video contentcomprises receiving a plurality of distinct streams of media contentfrom respective ones of the commentator applications; and wherein thecommented video content is provided to one or more servers of a contentdelivery network for delivery to one or more viewer devices.
 23. Theapparatus of claim 22 wherein the receiving and sending are implementedin a media server through interaction of the media server with webbrowsers of respective commentator devices that implement thecommentator applications.
 24. The apparatus of claim 23 wherein each ofthe web browsers implements an instance of a commentary mixer configuredto combine commentary information from its corresponding commentatorapplication with additional commentary information received fromrespective other ones of the commentator applications via the mediaserver.